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Abstract. Minimal representations of a real reductive group G are the 'small- 
est' irreducible unitary representations of G. We discuss special functions that 
arise in the analysis of L 2 -model of minimal representations. 



1. Introduction 

An irreducible unitary representation of a real reductive Lie group G is called 
minimal if its infinitesimal representation is annihilated by the Joseph ideal [11] 
in the enveloping algebra. Loosely speaking, minimal representations of G are the 
'smallest' infinite dimensional unitary representations. 

The Weil representation, known for a prominent role in number theory (e.g. the 
theta correspondence), provides minimal representations of the metaplectic group 
Mp(n,M). The minimal representation of a conformal group 5*0(4,2) appears in 
mathematical physics, e.g., as the bound states of the hydrogen atom, and inci- 
dentally as the quantum Kepler problem. In these classical examples the repre- 
sentations are highest weight modules, however, for more general reductive groups, 
minimal representations (if exist) may not be highest weight modules, see a pio- 
neering work of Kostant [21] for 5*0(4,4). 

In the last decade I have been developing a geometric and analytic theory of 
minimal representations with my collaborators, S. Ben Said, J. Hilgert, G. Mano, J. 
Mollers, B. 0rsted, and M. Pevzner, see [1, 6, 7, 8, 9, 12, 13, 15, 16, 17, 18, 19, 
20]. Among all, in this paper, we focus on 'special functions' that arise naturally in 
the L 2 -model of minimal representations. Needless to say, the interaction between 
special functions and group representations has a long history and there is extensive 
literature on this subject. A new feature in our setting for minimal representations 
is that the representation of the group is realized on the Hilbert space L 2 (E) where 
the dimension of a manifold S (see (2.1), or more generally Section 2.3 for the 
definition of S) is strictly smaller than the dimension of any nontrivial G-space 
in most cases. This means that G cannot act geometrically on S but there is 
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a natural action of G on L 2 (S). As a result, the Casimir element of a compact 
subgroup acts as a fourth-order differential operator. The indefinite orthogonal 
group G = 0(p + 1, q + 1) is the most interesting for this purpose in the sense that 
the group G itself contains two parameters p and q, and we shall highlight this case 
by giving occasionally some perspectives to other reductive groups. 

Acknowledgements: I first met Igor Frenkel when I visited Yale University in 
2009 to give a colloquium talk. It was then a surprising pleasure that Igor told me 
his recent theory on quaternionic analysis [3, 4, 5] with Libine uses some of my 
work [17, 19] on geometric analysis of minimal representations, which encourages 
me to develop further the analytic theory of minimal representations. 

I would like to thank the organizers P. Etingof, M. Khovanov, A. Kirillov Jr., A. 
Lachowska, A. Licata, A. Savage and G. Zuckerman for their hospitality during the 
stimulating conference "Perspectives in Representation Theory" in honor of Prof. 
Igor Frenkel's 60th birthday at Yale University, 12-16 May 2012. Thanks are also 
due to an anonymous referee for his/her careful comments. 

2. A generalization of the Fourier transform 

2.1. Algebraic characterization of Fourier transforms. We begin with 
an algebraic characterization of the Euclidean Fourier transform J 7 ^. Let Qj := Xj 
be the multiplication operators by coordinates, and Pj := -j=-£-. Then we have: 

PROPOSITION 2.1. Any continuous operator A on L 2 (R n ) satisfying 

A o Qj = Pj oA, A o P ■,= -Qj oA on S{R n ) (1 < j < n) 

is a scalar multiple of the Euclidean Fourier transform J-r™ . In particular, any such 
continuous operator A is unitary up to scaling. 

In place of K", let us consider the isotropic cone 

(2.1) S := {x e R p+ « \ {0} : x\ + ■ ■ ■ + x\ - x 2 p+1 x 2 p+q = 0}, 

equipped with a measure d[i = ^r p+q ~ z drdujdrj in the bipolar coordinates: 

(2.2) R+ x S^ 1 x S^ 1 ^ S, (r, u, rj) ^ (rw, rn). 

Remark 2.2. This cone H is a special case of the Lagrangian submanifold of a 
minimal real coadjoint orbit, denoted by the same letter 5, given in Theorem 2.10 
where we deal with more general reductive groups. 

Recall from [17] that the fundamental differential operators Rj (1 < j < p + q) 
on S are mutually commuting operators which are obtained as the restriction of 
the tangential differential operators 

£jXj n-(2E + p + q -2)-^- 

to H, where Ej = 1 (1 < j < p); = -1 (p + 1 < j < p + q), □ := YZt\ ^a£i (the 
Laplacian on R p > q ) and E := Y^a=i Xa ~$T (* ne Euler operator). Then we have 

Theorem 2.3 ([17, Theorem 1.2.3]). Suppose p + q is even, > 4. Then there 
exists a unitary operator F s on L 2 (E) satisfying the following relation for A: 

(2.3) A o Qj ; = Rj oA, Ao Rj = Qj o A on C C °°(S) (1 < j < p + q). 
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Conversely, any continuous operator A satisfying (2.3) is a scalar multiple of the 
unitary operator Ts ■ 

2.2. Unitary inversion operator F-b.- The similar nature of J-Rn and Ts. 
indicated in Proposition 2.1 and Theorem 2.3 is derived from the common fact that 
they arise as the unitary inversion operators in the L 2 -model (Schrddinger model) 
of minimal representations of real reductive groups Mp(n, M) and 0(p + l,q+ 1), 
respectively. 

To sec some more details of Theorem 2.3, let n, n, and [ be the Lie algebras 
generated by the operators Qi, Rj, and [Qi, Rj] (1 < i, j < p+q). Then g := n+l+n 
is isomorphic to o(p + l,q+ 1), and p = [ + n ~ (o(p, q) + R) x M. p+q is a maximal 
parabolic subalgebra of g. 

For p + q even, > 4, we proved in [19] that there exists an irreducible unitary 
representation tt of the group G := 0(p + l,q + 1) on the Hilbert space L 2 (E) of 
which the infinitesimal representation is given by Qj (the action of n) and Rj (the 
action of n), see also [17, Chapter 1]. We set 

(2.4) «,:=(£ °^eG. 

Geometrically, p is the Lie algebra of the conformal transformation group {0{p, q) ■ 
R>o) k W' q of the flat pscudo-Riemannian Euclidean space W' q , and w induces the 
conformal inversion of M. p ' q by the Mdbius transform. 

The unitary operators ir{g) are of simple form if g G G belong to the maximal 
parabolic subgroup P with Lie algebra p, namely, they are given by the multipli- 
cation of certain elementary functions on S and the translations coming from the 
geometric action of the Levi subgroup of P on S (sec [17, Chap. 2, Sect. 3]). In view 
of the Bruhat decomposition 

G = P II PwP, 

it is enough to find an explicit formula of the unitary operator n(w) in order to 
give a global formula of the G-action on L 2 (E). We call tt(w) the unitary inversion 
operator, and set 

(2.5) T E := n(w). 

We initiated in a series of papers [15, 16, 17] the following: 

PROGRAM 2.4 ([17, Program 1.2.5]). Use the unitary inversion operator T-~ 
for minimal representations as an analog of the Euclidean Fourier transform Tin, 
and develop a theory of 'Fourier analysis' on S. 

In the classical Schrddinger model of the Weil representation of the metaplec- 
tic group Mp(n,R) on L 2 (R n ), the unitary inversion operator is nothing but the 
Euclidean Fourier transform J 7 ^ (up to scalar of modulus one), see Example 3.1. 
We note that Mp(n, R) and 0(p + l,q + l) with p + q even are simple Lie groups 
of type C and D, respectively. 

The first stage of Program 2.4 is to establish a framework of the L 2 -model 
(Schrodinger model) of minimal representations, and to introduce the unitary in- 
version operator T~, with an algebraic characterization such as Theorem 2.3. In [8] 
we gave such a model and defined J-h by using Jordan algebras, see Section 2.3. In 
this case S is a Lagrangian subvariety of a minimal nilpotent coadjoint orbit and 
the resulting representations on L 2 (E) include a slightly wider family of unitary 
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representations than minimal representations (e.g. the full complementary series 
representations of 0(n, 1)). 

The second stage is to solve the following: 

Problem 2.5. Find an explicit formula of the integral kernel of T-.. 

We will discuss Problem 2.5 in Section 3. It is noteworthy that I. Frenkel and 
M. Libine have developed their original theory on quaterionic analysis in a series 
of papers [3, 4, 5] from the viewpoint of representation theory of the conformal 
group SX(2,Hc) — SX(4, C) and its real forms, and have demonstrated a close 
connection between minimal representations of various 0(p, g)'s and quaternionic 
analysis. For instance, the explicit formula of T~. for 0(3, 3) which was obtained in 
Kobayashi-Mano [17] is used in [5] for the study of the key operator (PIr in their 
notation) in the analysis of the space H R of split quaternions. 

The third stage and beyond will be based on the algebraic property (Theorem 
2.3) and analytic property (Problem 2.5) of the unitary inversion operator J 7 -. 
Among various, potentially interesting directions of the 'Fourier analysis' on S, 
here are some few topics: 

• A theory of holomorphic semigroups was given in Howe [10] for Mp(n, K) 
and in Kobayashi-Mano [16] for SO{n +1,2). 

• A deformation theory of the Euclidean Fourier transform J 7 ^ [1], e.g. an 



• A generalization of the classical Bargmann-Segal transform. See [9] in 
the case G/K is of tube type. 

Stage 1 already includes a solution for the Plancherel-type theorem of Ts- A 
natural but open question would be a Paley- Wiener type theorem of J 7 -: 

Question 2.6. Find an explicit characterization of J" H (C£°(E!)). 

Another important space of functions is an analog of Schwartz functions. For 
this we may consider: 

Definition 2.7 (Schwartz space on S). Let S(E) be the Frechet space of 
smooth vectors of the unitary representation of G on L 2 (E). 

This definition makes sense in a more general setting (see Theorem 2.10). By 
the general theory of unitary representations, we have: 

Proposition 2.8. J"e induces automorphisms of the Hilbert space L 2 (E) and 
the Frechet space <S(S). 




Jk :L 2 (~) ^ L 2 (S) 



{Plancherel type theorem) 



U 



U 



5(S) ^S(E) 



(Paley-Wiener type theorem). 



The following question is also open: 



Question 2.9. Find an explicit characterization ofS(E). 
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2.3. Schrodinger model of minimal representations. Suppose that V is 
a real simple Jordan algebra. Let G and L be the identity components of the 
conformal group and the structure group of the Jordan algebra V, respectively. 
Then the Lie algebra g is a real simple Lie algebra and has a Gelfand-Naimark 
decomposition g = n + 1 + n, where n ~ V is regarded as an Abelian Lie algebra, 
[ ~ stt(V) the structure algebra, and n acts on V by quadratic vector fields. 

Let O m ? n (c g*) be a (real) minimal nilpotent coadjoint orbit. By identifying g 
with the dual 0*, we consider the intersection 



the Kirillov-Kostant-Souriau symplectic form. There is a natural L-invariant 
Radon measure on S, and we write £ 2 (S) for the Hilbert space consisting of square 
integrable functions on S. 

Theorem 2.10 (Schrodinger model [8]). Suppose V is a real simple Jordan 
algebra such that its maximal Euclidean Jordan algebra is also simple. Among all 
such Jordan algebras V, we exclude the case where V ~ M. p ' q with p + q odd (see 
Examples 2.13 and 2.1 1). 

1) For an appropriate finite covering group G of G there exists a natural unitary 
representation of G on L 2 (S). It is irreducible if and only ifB. is connected. 

2) The Gelfand-Kirillov dimension of ir attains its minimum among all infinite 
dimensional representations of G, i.e. DIM(7r) = ^dimO^. 

3) The annihilator of the differential representation dn is the Joseph ideal in the 
enveloping algebra U(qc) if V is split and gc is not of type A. 

The simple Lie algebras g that appear in Theorem 2.10 are categorized into 
four cases as below: 



Example 2.11. If V is a Euclidean Jordan algebra, then G is the automorphism 
group of a Hermitian symmetric space of tube type and the corresponding Lie 
algebra g is listed in (2.7). In this case H consists of two connected components, 
and the resulting representation tt is the direct sum of an irreducible unitary highest 
weight module and its dual. 

Remark 2.12. In the case (2.9) the complex minimal nilpotent orbit 0^; n does 
not meet the real form g, and there does not exist an admissible representation of 
any Lie group with Lie algebra g. In particular, the representation tt in Theorem 
2.10 is not a minimal representation but still one of the 'smallest' infinite dimen- 
sional representations in the sense that the Gelfand-Kirillov dimension attains its 
minimum. 

Example 2.13. Let V = Sym(n,R). Then g =sp(n,R) and 




(2.6) 
(2.7) 
(2.8) 
(2.9) 



s((2fc,M),so(2/c,2fc),so(p+ l,q+ l),e 7(7) , 
sp(k, R),su(fc, fc),so*(4fc),so(2, k), e 7( _ 2 5), 
sp(fc,C),s[(2fc,C),so(4fc,C),so(fc + 2,C),e 7 (C), 
sp(fc,fc),su*(4fc),so(fc, 1). 



(2.10) 



S = {X € M(n,R) : X = ^rankX = 1}. 
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Let S+ := {X e V n 0°? n : Traced > 0}. Via the double covering map (folding 
map) 

R"\{0}^S+, v^+v f v 
we can identify the representation on L 2 (S + ) with the even part of the Schrodinger 
model on L 2 (R") of the metaplectic group Mp(n,R) [2, 10]. See [8] for the re- 
alization of the odd part of the Weil representation in the space of sections for a 
certain line bundle over S+. 

Example 2.14. Wc define a multiplication on W+i = M ® W+i- 1 by 

P p+q 

{xi,x') ■ {yi,y') := {x 1 y 1 - ^x.y, + ^ Xiy^xiy' + yix'). 

i=2 i=p+l 

The resulting Jordan algebra is denoted by M. p ' q (by a little abuse of notation). It is 
a semisimple Jordan algebra of rank two, and its conformal algebra is o(p+l, q + 1). 
Suppose now V = W' q with p + q even. Then 5 in Theorem 2.10 coincides with the 
isotropic cone given in (2.1). For q = 1, V is an Euclidean Jordan algebra, and S 
consists of two connected components according to the sign of the first coordinate 
xi, i.e. the past and future cones. For p, q > 2, V is non-Euclidean, S is connected, 
and our representation it on £ 2 (S) is the same as the Schrodinger model of the 
minimal representation of 0(p + l,q + 1) constructed in [19, Part III], which is 
neither a highest nor a lowest weight module. 

Remark 2.15. There is no minimal representation for any group with Lie 
algebra o(p + l,q+l) with p + q odd, p, q > 3 (see [23, Theorem 2.13]). 

3. Unitary inversion operator F-b. 

By the Schwartz kernel theorem, the unitary inversion operator Ts. can be given 
by a distribution kernel K(x,y) £ D'(S x S), namely, 

Tb.u{x) = f K(x, y)u(y)dfj,(y) for all u e C C °°(S). 

Problem 2.5 asks for an explicit formula of K(x,y). In the setting of Theorem 
2.10, we can generalize the definition (2.5) of T~, by taking to to be a lift of the 
conformal inversion on V, see [9]. So far, Problem 2.5 has been solved for minimal 
representations in the following two cases: 

Case A. G is the biholomorphic transformation group of a Hcrmitian symmetric 
space of tube type ([9]). 
CaseB. G = 0(p + l,q+l) (Theorem 3.3). 
Case A includes the following earlier results: 

Example 3.1. 1) G = Mp(n,R), S = M™, 7r = the Weil representation. 

(2ir) 2 

In this case T~. is the Euclidean Fourier transform Tw* up to a phase 
factor c with |c| = 1, see [2]. 
2) G = SO(p + 1,2), S is the light cone (q = 1 in (2.1)), tt = the highest 
weight representation of the smallest Gelfand-Kirillov dimension and of 
the smallest isf-type. 

K(x,y) =cJ v _ 1 {2^j2{x 1 y)) 
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where J\(t) := (|) J\(t) is a renormalization of the J-Bessel function 
([15])- 

3.1. Mellin Barnes type integral expression. In [17] we brought an idea 
of the Radon transform in the analysis of the unitary inversion operator J 7 - for 
minimal representations. Recall that the Euclidean Fourier transform J^n can be 
written as the composition of the one-dimensional Fourier transform and the Radon 
transform (plane wave decomposition). We can generalize this decomposition to 
the unitary inversion operator Ts. for the minimal representations and some small 
representations on L 2 (S) given by Theorem 2.10, namely, there exists a distribution 
<fr(t) of one variable such that the distribution kernel K(x, y) of T-= is of the following 
form: 

(3-1) K(x,y) = *((x,y)), 

where ( , ) is some (natural) bilinear form of the ambient space V. Thus Problem 
2.5 reduces to find a formula of $(t). 

Example 3.2. In Example 3.1 (2) we have seen $(i) = cJ P -3 (2V2t) when 
G = 0(p + 1, 2). Therefore, J*e reduces to the Hankel transform composed with a 
'Radon transform' on S in this case. 

The formula of is more involved for G = 0(p+ 1, q + 1) with p, q > 2 as the 
corresponding minimal representation is not a highest weight module and &(t) is 
not always locally integrable (see Question 3.4 below). An explicit formula of 
in this case can be given in terms of 'Bessel distributions' [17, Theorem 5.1.1]. Here 
we give an alternative expression of 3>(t), namely, by using a distribution- valued 
Mcllin-Barnes integral. 

We define a distribution of t with meromorphic parameter A by 



Here the Riesz distribution (2t)+ is defined as a locally integrable function on R by 

(2*)i 



(2t) x t > 
t < 



for Re A > — 1, and is extended as a distribution by the meromorphic continuation 
on A e C. Let m := |(p + q — 4), and L m be a contour starting at 7 — ioo, passes 
the real axis between (— m — 1, —to) and ends at 7 + ioo when 7 > — 1. We define 
distributions $ p ' 9 (£) by a distribution- valued Mellin-Barnes integral: 

!J Lo b(\,t)d\ (CaseA-1), 
j L ° b(X,t)d\ (Case B-l), 

OS^ + HfegW (CascB-2), 
according to the following three cases: 
Case A-l. p = 1 or q = 1, 
Case B-l. p, q > 1 and both odd, 
Case B-2. p, q > 1 and both even. 

Then Q p ' q (t) is independent of the choice of the contour and 7 under the above 
mentioned constraints. 

Let ( , ) be the (positive definite) inner product on W' q . Then we have 
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Theorem 3.3 ([17, §6.2]). For G = 0(p + l,q + l) with p + q even, > A, the 
kernel K(x 7 y) of the unitary inversion Ts is given by 

K(x,y) = c p , q $™({x,y)) 

for some constant c p , q . 

3.2. Local integrability of the kernel. The kernel of the Euclidean Fourier 
transform F^n is given by e~ l ( x, & , which is locally integrable. We may ask to which 
extent this analytic feature remains to hold. To be more precise, let be the 
distribution on R as in (3.1). We ask 

Question 3.4. When is $(i) locally integrable? 

For a Euclidean Jordan algebra V we proved in [9] that $(t) is locally integrable. 
See (2.7) for the list of the corresponding conformal Lie algebras 0. 

For G = 0(p + l,q + 1) with p + q even > 2, the Mellin-Barnes type integral 
formula (Theorem 3.3) leads to the following proposition (see [17, Theorem 6.2.1]): 



PROPOSITION 3.5. We have the identities modulo L\ oc {R,r p+q 3 dr). 
<P p - q (t) = < 



'0 (CascA-1), 
eiE^o 1 ^i-V-i). *'^) (CaseB-1), 
UTZo 1 ^T^yi- 1 - 1 (CaseB-2), 



for some nonzero constants C\, c 2 . Here m = | (p + q — 4) . 

Thus we have a complete answer to Question 3.4 in this case: 

Corollary 3.6. <&(t) is locally integrable if and only if Q = o(p+ 1, 2), o(2, q + 
1) or o(3,3) ~sl(4,K). 

We note that the minimal representations for q = o(p + 1, 2) or o(2, q + 1) are 
highest (or lowest) weight modules, whereas minimal representations do not make 
sense for q = o(3,3) which is isomorphic to sl(4, R). (Recall that the Joseph ideal 
is defined when jjc is not of type A.) 

The delicate answer indicated in Corollary 3.6 is closely related to the regularity 
of the 'Radon transform' on S. To be more precise, the Radon transform on S is 
defined as the integral over the codimension-one submanifold 

(x,y)=t in S. 

which collapses when t = 0. Accordingly, the Radon transform 

(nu)(x,t) = [ u(y)5((x,y) - t)dy 

has a better regularity as t tends to 0. (On the other hand, the asymptotic behavior 
as \t\ — > oo is similar to the Euclidean case.) The singular part of $ pq (t) in 
Proposition 3.5 fits well with the behavior of 1Zu(x, t) as t tends to 0. 

Question 3.4 is open for minimal representations without highest weights except 
for the case G = 0(p + l,q+ 1). 
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4. Fourth order differential equations 

4.1. Gaussian kernel and minimal K- type. The Euclidean Fourier trans- 
form J-Rn is of order four, and therefore its eigenvalues in i 2 (R") are among 
{±1,±\/— 1}. An important eigenfunction with eigenvalue 1 is the Gaussian kernel 
e -2ll x ll , namely, 

j- R „( e -|INI 2 ) =e -HNI 2 . 

Thus the Gaussian kernel e^" 2 '" 2 is a square integrable function on R n satisfying 
the following property: 

(4.1) f = / and / is 0(n)-invariant. 

Next let p > q > 1, p + q even, and we consider the isotropic cone S in M. p ' q 
as in (2.1). The unitary inversion operator !F-~ of the minimal representation tt of 
0(p + 1, q + 1) is of order two because Ts. = n(w) and w 2 = I p+q (see (2.4)), and 
therefore its eigenvalues in L 2 (E,) are either 1 or —1. An important eigenfunction 
of T-z is ^i( g _ 2 )(2||a;||) where K\{t) := (^)~ x K\(t) is a renormalization of the K- 
Bessel function. This is a square integrable function on S satisfying the following 
property: 

(4.2) T s f = ±f and / is 0(p) x 0(q)-invariant. 
To be more precise, 

-p_j = \f ifp-? = mod 4, 

1 — / if p — q = 2 mod 4. 

Example 4.1. For p = 3 and q = 1, we have 

j- s(e -2MI ) = _ e -2MI ; 

because 

(4.3) K_ i (t) = ^e- t . 

The function e 2 1 1 ^ 1 1 arises as the wave function for the hydrogen atom with 
the lowest energy. 

in ii 2 

From the view point of representation theory, the Gaussian kernel e~2" x " 
generates the minimal 7^-type of the Weil representation of Mp(n,M.), whereas the 
function K i( g _ 2 )(2||x||) generates that of the minimal representation of 0(p+l, q + 
1) realized in L 2 (E). 

4.2. The Mano polynomial. We recall a classical fact that the Hermite 
polynomials form an orthogonal basis for the radial part of the Schrodingcr model 
L 2 (R n ) of the Weil representation e.g. [2, 10], whereas the Laguerre polynomials 
arise in the minimal representation of the conformal group SO (4, 2). The bottom 
of the series correspond to what we have discussed in Section 4.1. 

We notice that these two minimal representations are quite special, namely, 
they are highest weight modules. However, for more general reductive groups, 
minimal representations are not always highest weight modules, and we need new 
'orthogonal polynomials' and 'special functions' to describe a natural basis of func- 
tions satisfying (4.2) or alike. 
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For fj, e C \ {-1, -2, -3, • • • } and I e N, the Mano polynomials {M?' e (x)} j€ n 
arc defined by 

(4.4) Mf{x) := rC?+/i + ^ ) 1 |1 G"'(t, *), 
where the generating function G M,i (i,x) is given by 

(4.5) G»'{t ,x) := iH-—^/^ (-^—]k t+1 (-JL—]. 

Here 7 a (z) := (§)~ a / Q (z) and ^(z) = (^)^ a K a (z) denote the rcnormalized I- 
and -fT-Bessel functions. With this normalization, the polynomial Mj' i (x) is of the 
following top term: 

(—1)3 

M? ,e (x) = : — x j+i + lower order terms. 

r- 

Example 4.2 (Special values of the Mano polynomial). 

(1) The bottom of the series with j = is related with the JT-Bessel functions 
with half-integer parameter: 

M-\x) = n-h^e*K i+ i(z) (= £ ||^| A 

(2) The polynomials Mf' l {x) for £ = reduce to the Laguerre polynomials 



r*o M _ r (. w r(n + »/ + !) V- r_i^fc ^ £ 



„fc 



M rW = L ?W (=^ r ^^(-l)« WF ^ rrT ^). 

fe=0 

(3) The function Mj l ' e (x) is not a polynomial when £ ^ N, but it is convenient 
to include the negative integer case. In particular, for I = —1, it follows 
from [6, Corollary 5.3] and [7, Lemma 3.2] that Mj~ 1 (x) is essentially 
the Laguerre polynomial: 

xM?'- 1 (x) = L'?(x) 

Many of the classical orthogonal polynomials are obtained as eigenfunctions 
of self-adjoint differential operators of second-order, but the Mano polynomials 
are obtained as those of fourth-order. Indeed this is a requirement from 
representation theory because the Casimir operator (for a compact subgroup) acts 
as a fourth-order differential operator on S. To see this we may recall that the 
Lie algebra n acts as a second order differential operator (e.g. the fundamental 
differential operator Rj in (2.3)). 

We begin with a second order differential operator on R 

, d _ „ x , . d x , , x , n 

^ : =(^+M-2^1--)(^+M-2)-(2) 2 ' 

and introduce a fourth order differential operator 
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Proposition 4.3 ([7]). (1) (Differential equation) The Mano polynomial M?' (x) 
is an eigenfunction ofV^^: 

(4.6) V^u = j(j + f i + l)u. 

(2) (Completeness) If fj, > 21+1 is an odd integer, then {Mj' l }j e ^ forms an 
orthogonal basis of L 2 (R + , x tl ~ 2l e~ x dx). 

There are two proofs of an explicit formula for the i 2 -norm of Mj' i (x), see [7, 
Theorem 2.4] and [18, Corollary 4.1]. 

4.3. if-finite vectors for minimal representations. We discuss a rela- 
tionship between the Mano polynomials and minimal representations. Let 5 be the 
isotropic cone in M. p ' q as in (2.1) with p + q > 4 and even. 

We recall that the space of fcth spherical harmonics 

n k (R n ) := {if G C^iS"- 1 ) : A S r,-i<p = -k(k + n- 2)<p} 

is a finite dimensional vector space, on which the orthogonal group 0(n) acts 
irreducibly by rotations. Then we have an irreducible representation of K := 
0(p + l) x 0(q + l) on 

V 3 :=W(W +1 )®W +2 ^{M. q+1 ), j = 0,1,2,- ■■ . 

Let L 2 (E)k be the underlying (jj, i^-module of the minimal representation L 2 (E) 
of G = 0(p + l,q+l). Then by [19], L 2 (E)k is isomorphic to the multiplicity-free 
sum ffi^Lo^- 7 as if-modules. We write L 2 (E)j for the corresponding i^-irrcducible 
subspace of i 2 (S). 

With the notation of Section 2.2, K n P ~ 0(p) x 0(q) x Z 2 . We set M := 
O(p) x 0(g) . The representation of the whole group G (or even the maximal compact 
subgroup K) on L 2 (E) does not come from the geometric action of G on S, but 
the action of the subgroup M is given by rotation in the argument and the bipolar 
coordinates (2.2) respect the M-action. Correspondingly, it is not straightforward 
to find explicit X-finitc vectors in L 2 (E), whereas the M-invariant functions only 
depend on the radial parameter r e M + . We identify the space L 2 (E) M of M- 
invariants in L 2 (E) with L 2 (R +1 (l/2)r p+q ~ 3 dr) as we saw in Section 2.1. 

Proposition 4.4 ([6, Section 8]). Let G = 0(2m,2n). For every j E N, the 
subspace L 2 (E)j ni 2 (S) M is one- dimensional and given by the Mano polynomials: 

uf n {x) := x- 2n+3 e- x M* m - 3 ' 2n - 3 (x). 
Here x — 2r in the bipolar coordinates (2.2) o/S. 

Therefore the functions {u™ ,n {2r)}j^ give a basis of functions / satisfying 
(4.2). To be more precise, we have 

m n = \u™' n if n-m = j mod 2, 

~ Uj " \ -uf' n iin-m^j mod 2. 

Example 4.5. 1) The bottom parameter j = explains that Ki^ q _ 2 ^(2r) 
generates the minimal K-type (Section 4.1). 

2) The case n = 1 recovers the classical fact that the Laguerre polynomials generate 
every if-finite vectors for the minimal representations of SO(2m, 2). 
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The following remarkable observation is a consequence of Example 4.2 (3) and 
Proposition 4.4: 

Observation 4.6. Let G = 0(2m, 4) with m > 2. Then any K-type of tt is 
given by using Laguerre polynomials. 

We notice that the minimal representations of 0(2m, 4) are not highest weight 
modules. It would be interesting to find a different proof for Observation 4.6, 
possibly in connection with other areas of mathematics. 

In contrast to Proposition 4.4 for G — 0(2m,2n), il-finite vectors in F 2 (S) 
cannot be expressed by elementary functions for G — 0(2m + 1, 2n + 1). For this 
we introduced in [6] a family of real analytic functions {Aj' v (x)}j e ^ on M + by the 
generating function: 

fcrfi) - — J_^.(Je_)#,,(_e_). 

Then we have the following Proposition: 

PROPOSITION 4.7 ([6, Corollary 8.2]). Let G = 0(p + 1, q + 1) withp + q even, 
> 2. Then 

L 2 (E)j n L 2 (0 M = CA^ 2 ' ? - 2 (2r) 

/or all j'eN. 

COROLLARY 4.8. For \i> v > — 1, n, v e Z w^/i [i = v mod 2 and (/i, i/) 7^ 
(—1,-1), {A^ t,!y (a:)} :/e N forms an orthogonal basis in L 2 (J& +1 x^ +v+1 dx). 

Remark 4.9. The function A^'^ can be expressed by elementary functions 
when v e 2Z+ 1: 

A^ +1 0r) - 2Mr0 ' + ^ ) x- 2 ^ 1 e ^Mf £ (2x). 

Thus Proposition 4.7 includes Proposition 4.4 as a special case. 

Remark 4.10. The indefinite orthogonal group 0(p,q) has two parameters p 
and q, and the corresponding special functions are expected to be the most general. 
In fact, for other minimal representations and some small representations given in 
Theorem 2.10, analogous results of Proposition 4.7 remain true by a specific choice 
of the parameters \x and v. 
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